Modeling Mobile Interface Tappability Using Crowdsourcing and Deep Learning

Tapping is an immensely important gesture in mobile touchscreen interfaces, yet people still frequently are required to learn which elements tappable through trial and error. Predicting human behavior for this everyday can help app designers understand aspect of the usability their apps without having run a user study. In chapter, we present approach modeling tappability interfaces at scale. We conducted large-scale data collection interface over rich set using crowdsourcing computationally investigated variety signifiers that use distinguish versus not elements. Based on dataset, developed trained deep neural network predicts how likely will perceive element as tappable. To demonstrate capability model, TapShoe, tool automatically diagnoses mismatches between each perceived by user—predicted our intended or actual state specified developer designer. Our model achieved reasonable accuracy: mean precision 90.2% recall 87.0%, matching perception identifying UI The TapShoe were well received via informal evaluation with 7 professional interaction designers.